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MICHAEL BARR 

Co-Founder/CTO, Barr Group 


Electrical Engineer (BSEE/MSEE) 

Experienced Embedded Software Developer 
Consultant & Trainer (1999-present) 

■ Embedded software process and architecture improvement 

■ Various industries (e.g., medical devices, industrial controls) 



Former Adjunct Professor 

■ University of Maryland 2000-2003 (Design and Use of Operating Systems) 

■ Johns Hopkins University 2012 (Embedded Software Architecture) 

Served as Editor-in-Chief, Columnist, Conference Chair 


Expert witness (software patents/copyrights, product liability) 


Author of 3 books and 70+ articles/papers 
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The Embedded Systems Experts 
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Skills Training 
Engineering Guidance 
Product Development 


Skills Training 

In this age of quickly changing 
technology everyone needs 
training from time to time. We 
provide skills training online and 
on-site to keep you up to date 
with the latest in embedded 
systems technology. 



Barr Group helps companies make their 
embedded systems safer and more secure. 

@barrgroup #embedsys #safety #security 

BARR 
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SAFETY & EMBEDDED SOFTWARE 



Past, present, future of lethal software... 



\ 



SAFETY PAST 
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Patriot Missile 

■ Failed to track a Scud 

Therac-25 

■ Massive overradiation 

Combined cost 

■ 30 dead 

■ >100 injured 
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http://jasonlove.com/funny-ca rtoons/search-viewer.aspx?id=1125 
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PATRIOT MISSILE FAILURE 



_ 


GAO: Software Problem Led 
to System Failure at 
Dhahran, Saudi Arabia 


February 25, 1991 

■ 28 U.S. soldiers dead; 98 wounded 

■ Single deadliest incident for U.S. 
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THE PATRIOT SOFTWARE BUG 


10001 


Two versions of system time 

■ Timer chip integer representation 

■ Software fixed-point binary format 

7.5s: 000000000000000000000111.100000000000000000000000 


Increasing inaccuracy... 


3. Track Action • Only Range Gated 
Portion of Beam Processed 



uptime (h) 

error (s) 

shift (m) 

1 

.0034 

7 

8 

.0275 

55 

20 

.0687 

137 

100 

.3433 

687 
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GAO Report : https://www.fas.org/spp/starwars/gao/im92026.htm 









NOTEWORTHY QUOTES 
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Brig. Gen. Neal, U.S. Command (+2 days) 

■ “looks like this [Scud] broke apart in flight... [thus] 
wasn’t in the parameters where it could be attacked” 

Col. Garnett, Patriot Program Director (+4 months) 

■ “an anomaly that never showed up in thousands of 
hours of testing” 
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Sources : Contemporaneous New York Times articles; available online. 
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THERAC-25 SYSTEM OVERVIEW 


THERAC 25 SET-UP 

i- 

X*ay source 


VT IOC 



POP 11 


\7 


movable 
lurxjslen 
screen 


▼ 

patient 


Installations in 5 U.S. and 6 Canadian facilities 

■ Thousands of treatments as intended, but... 
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Images : http://hci.cs.siue.edu/NSF/Files/Semester/Week13-2/PPT-Text/Slidel 3.html 

















































6 MASSIVE OVER-DOSES 
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Kennestone Regional Oncology Center, June 1985 


Ontario Cancer Foundation, July 1985 


Yakima Valley Memorial Hospital, December 1985 


East Texas Cancer Center, March 1986 


East Texas Cancer Center, April 1986 


Yakima Valley Memorial Hospital, January 1987 
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Source : http://sunnyday.mit.edu/papers/therac.pdf 
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ONE OF THE THERAC-25 BUGS 


Hkeper 


Task 


Lmtchk ( ) 

If Class3=Q 

Then do not enter Chkcol 

If Class3 is not 0 ^ ^ 

Then enter Chkcol 


Chkcol ( ) 

If upper collimator 
inconsistent with treatment 
then set bit 9 of F$mal 



V 

\ 

J 



V 

\ 

V 



\ 


\ 


J 




Source : http://sunnyday.mit.edu/papers/therac.pdf 
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Treat 
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Task 
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Set Up Test ( ) 

During Set 

Increment Class 3 on each cycle " 

^ Check F$mal 

If F$mal=0 system is consistent 
" ' then set Tphase=2 for Set Up Done 


a. 


V 


ciass3 global 
rolls every 256 



















































NOTEWORTHY QUOTES 
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AECL Letter (Feb '86, in response to 3 rd incident) 

■ “After careful consideration we are of the opinion that 

this [injury] could not have been produced by any 
malfunction of the Therac-25” 

“no other instances of similar [patient] damage” 

reddening of the skin) in a parallel striped pattern on her right hip. 


Quality Assurance Manager (to User's Group) 

■ Therac-25 software was tested for “2,700 hours” 
Under questioning: “2,700 hours of use” 
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Source : http://sunnyday.mit.edu/papers/therac.pdf 
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Underestimation of software risks can be deadly 


Hazard Analysis. In March 1983, AECL performed a safety analysis on 
the Therac-25. This analysis was in the form of a fault tree and apparently 
excluded the software. According to the final report, the analysis made 
several assumptions about the computer and its software: 

1. Programming errors have been reduced by extensive test¬ 
ing on a hardware simulator and under field conditions on 
teletherapy units. Any residual software errors are not in¬ 
cluded in the analysis. 

2. Program software does not degrade due to wear, fatigue, or 
reproduction pro cess. 


More: Leveson, IEEE Computer, Jul 1993 
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Source : http://sunnyday.mit.edu/papers/therac.pdf 













SAFETY PRESENT 
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Some systems are “safety-critical” 


Exposure to low probability events... 

■ Random events in the electronics, 

■ Bugs latent in the software, and/or 

■ Unforeseen gaps through fail-safes 

Testing cannot prove absence of bugs/gaps... 

■ Therefore, system safety only as by design 

BARR 
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AUTOMOTIVE SOFTWARE TRENDS 
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Image : http://gearheads.org/understanding-the-brain-at-the-heart-of-your-car/ 
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AUTOMOTIVE SOFTWARE RECALLS 


RECALLS 


Honda recalls nearly 350k Odyssey minivans 
over unintended braking 





The issue revolves around a combination of parts and software that have been reported to cause 
to brake hard and unexpectedly, without illuminating the brake lights. Imagine driving behind one of these 
vehicles when the malfunction occurs and you can easily understand how an unexpected rear-end collision 


General Motors recalls 370,000 GM, Chevy 
pic kups with engine fire risk 


The trucks are only supposed to use two cylinders 
when idling, but a software glitch is causing them to 
idle with most of their cylinders. This can cause 
exhaust components to overheat, and hence 
potentially catch fire. ^CHRISTIAN SCIENCE 
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TOYOTA & UNINTENDED ACCELERATION 


Toyota adds “electronic throttle” -2002 models 


NHTSA investigates “UA” complaints (5 times) 


Models at Issue 

End Date 

Recall? 

2002-2005 Camry/Solara, Lexus ES 

Jan '06 

none 

2002-2006 Camry/Solara 

Apr '07 

none 

2007-2008 Camry, Lexus ES 

Sep '07 

all-weather floor mat 

2006-2007 Tacoma 

Aug '08 

none 

2004 Sienna 

Jan '09 

trim clip 


Then a high profile crash... 
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Timeline : https://www.consumerreports.org/cro/news/2010/02/timeline-of-toyota-acceleration-investigations/index.htm 








CHP Officer, F amily Killed in Crash 


A|911 calT|made minutes before the accident said the car's accelerator was stuck 


By Rory Devine, Mari Payton and R. Stickney | Tuesday, Sep 1,2009 


View Comments () | Email | Print 


SAN DIEGO 


Source : http://www.nbcsandiego.com/news/local/CHP-Officer-Family-Killed-in-Crash-56629472.html 


/** 









An image taken from the air shows the vehicle resting in the brush just off the road. 


“Saylor” 

28 Aug ’09 
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UNINTENDED ACCELERATION 


What is unintended acceleration? 

■ Acceleration the driver did not purposely cause 

1 In this report, “unintended acceleration” refers to the occurrence of any degree of acceleration that the vehicle 
driver did not purposely cause to occur. Contrast this with the term “sudden acceleration incident,” which refers to 
“unintended, unexpected, high-power accelerations from a stationary position or a very low initial speed 
accompanied by an apparent loss of braking effectiveness.” An Examination of Sadden Acceleration, DOT-TSC- 
NHTSA-89-1 at v. As used here, unintended acceleration is a very broad term that encompasses sudden acceleration 
as well as incidents at higher speeds and incidents where brakes were partially or fully effective, including 
occurrences such as pedal entrapment by floor mats at full throttle and high speeds and incidents of lesser throttle 
openings at various speeds). 

- 3 - 


Loss of driver control of engine power 

■ Avery dangerous vehicle malfunction! 


19 
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Source : http://www.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pdf (“NHTSA”), p. vi 
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ELECTRONIC THROTTLE CONTROL 




accelerator 


Engine Software 


Driver Controls 


Combustion 



throttle control 



spark 
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POSSIBLE SOURCES OF ACCELERATION 


Mechanical 

■ Pedal entrapped by floor mat 

■ Sticky pedal by internal defect 

■ Stuck throttle valve 


-30% of models recalled 


Driver error 

■ “Pedal misapplication” 


Software malfunction 
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Engine Software 



accelerator 


Driver Controls 

Combustion 

(Not Working) 

(Working) 



fuel 


throttle ("stuck") 


air 


/ 


spark 
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TOYOTA’S HIGH COMPLAINT RATE 


Complaints jump after “electronic throttle" 


■ NHTSA data 2004 vs. 2000-2003 

All UA complaints -2,000 (vs. 1,200-1,400) 
Toyota’s percentage -20% (vs. 4-7%) 


m 

■ Toyota 
+300% 


Complaint Statistics : http://democrats.energycommerce.house.gov/Press_111/20100222/ 
Detailed.Timeline.and.Background.of.NHTSA.Actions.Regarding.Toyota.Sudden.Acceleration.pdf 


Could driver errors explain the jump? 

■ Expect driver errors -even across makes 

■ Why such a big increase w/in Toyota? 
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By CBS NEWS AP May 25 , 2010 , 7:08 PM 


<DCBS 

mone y match 

^ .com 


Toyota "Unintended 
Acceleration" Has Killed 89 



The National Highway Traffic Safety Administration said that from 2000 to mid- 
May, it had received more than 6,200 complaints involving sudden acceleration in 
Toyota vehicles. The reports include 89 deaths and 57 injuries over the same 
period. Previously, 52 deaths had been suspected of being connected to the 

Problem. Source: http://www.cbsnews.com/news/toyota-unintended-acceleration-has-killed-89/ 












Statistical analysis and slide by R.A. Whitfield to NAS’ Transportation Research Board (Oct 2010) 

How likely is it that these factors... 


Vehicle Factors: 

Floor mats Recalls of 
Sticky pedals some cars. 
Pedal placement 
Gated gear shift pattern 
Ignition switch design 


Driver Factors:_ 

Mass hysteria 
Fraud 
Old age 
Youth 

Inexperience 
Incompetent drivers 


Environmental/Usage 

Factors 


Factors held 
-constant. 


...explain these results when controlling 
for make/model and years in service? 


Camry (see page 42) 


S.00 

4.00 

3.00 

2.00 

1.00 

0 


Without ETCS-i 


With ETCS-i 


ES 300 Series (see page 43) 



4.67 

10.00 


8.00 


6.00 



4.00 


2.00 

/■v 



Without ETCS-i 


With ETCS-i 


Tacoma (see page 44) 


3.00 

2.50 
2.00 

1.50 
1.00 
0.50 

0 


UA complaints to NHTSA “pre-Saylor”, in 1 st year of model sale per 100K. 




286 









0.75 







Without ETCS-i With ETCS-i 


Source : http://onlinepubs.trb.org/onlinepubs/UA/101011Whitfield.pdf 
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NASA 


“THE NASA REPORT” 




At NHTSA’s request 

■ Published Feb ’11 

Lots of redactions 

■ Especially re: software 

I can’t talk about them! 

Some flaws found... 

...but not “the cause” 


25 Copyright 2014 Barr Group. All rights reserved. 

Public (Redacted) NASA Report : https://www.nhtsa.gov/UA 



NASA Engineering and Safety Center 

Technical Assessment Report 

Version: 

1.0 

Title: 

National Highway Traffic Safety Administration 

Toyota Unintended Acceleration Investigation - 
Appendix A 

Page #: 

17 of 134 
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NASA KEY NASA STATEMENTS 


Because proof that the ETCS-i caused the reported UAs was not found does not mean it could 

not occur. However, the testing and analysis described in this report did not find that TMC 
“ Due to system complexity which will be described and the many possible electronic hardware T 
and software systems interactions, it is not realistic to attempt to “prove” that the ETCS-i cannot 
cause UAs. Today’s vehicles are sufficiently complex that no reasonable amount of analysis or 
testing can prove electronics and software have no errors. Therefore, absence of proof that the 
ETCS-i has caused a UA does not vindicate the system.| From calendar year 2005 to 2010 TMC 


The NESC team identified two hypothetical ETCS|-i failure mode scenarios (as opposed to non¬ 
electronic pedal problems caused by sticking accelerator pedal, floor mat entrapment, or operator 
misapplication) that could lead to a UA without generating a diagnostic trouble code (DTC): 
specific dual failures in the pedal position sensing system and a systematic software malfunction 


The second postulated scenario is a systematic software malfunction in the Main CPU that opens 
the throttle without operator action and continues to properly control fuel injection and ignition.| 
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Source: http://www.nhtsa.gov/staticfiles/nvs/pdf/NASA-UA_report.pdf (“NASA”), pp. 15-20 
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LAWSUITS... 


Toyota had to produce source code and design docs 

■ Lawyers for lead U.S. plaintiffs brought in Barr Group 


Plaintiff 

Court 

My Role 

Status 

Amount 

Saylor 

CA 

* 

settled Feb Ml 

$10M 

Van Alfen 

U.S. 

report Jul ’12 

settled Dec M2 

private 

U.S. Class 

U.S. 

report Jul '12 

settled Dec M2 

up to $1.5B 

St. John 

U.S. 

report Apr M3 

in talks now 


Bookout 

OK j 

testimony Oct M3 

jury trial Oct M3 



* Saylor (and some other early plaintiffs) did not look into software. 
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OUR REVIEW OF TOYOTA’S SOFTWARE 


Access to Toyota’s engine source code 

■ Seven Toyota and Lexus models x -2002-2010 model yrs 

Approximately 18 months of calendar time 

■ By an experienced team of embedded practitioners 

■ Building on NASA’s earlier work; digging deeper 

Access to more software/code (per vehicle) 

Bottom-up focus on software details 
Simulation and in-vehicle testing 
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SOURCE CODE CONFIDENTIALITY 


Custom-built room 

■ No Internet 

■ No phones 

Layered security 

■ Guard station 

■ More... I can’t say! 

At a secret address... 

29 Copyright 2014 Barr Group. All rights reserved. 



Photo: Not of the actual source code room. 

















BOOKOUT FACTS 


Single-vehicle Sep 2007 accident 

■ On exit ramp from US-69 South 

Near Eufaula Lake, Oklahoma 

Vehicle 

■ 2005 Toyota Camry (4-cylinder) 

Two occupants 

■ Driver Jean Bookout: seriously injured 

■ Passenger Barbara Schwarz: died later 

Witness to driver’s braking 

30 Copyright 2014 Barr Group. All rights reserved. 




BOOKOUT RECONSTRUCTION 


Speed estimates 

■ Skid start ~50mph 

■ At impact ~25mph 

Agreed she braked - 

■ Parking brake too? 

150’ skid mark - 

■ Way too long! 
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OPEN THROTTLE DEGRADES BRAKING 


Proof via Saylor crash 

■ “Pedal stuck” 

Top speed ~ 120 mph 

■ Healthy male age 45 

Couldn’t stop by braking 



Consumer Reports 

■ “80 miles an hour. I am powerless to slow this vehicle” 


■ After pumping... “even one time ... 
impossible to stop the vehicle. ” 
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Consumer Reports Video : http://youtu.be/VZZNR903xZM 
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BRAKE VS. THROTTLE DATA POINTS 


At large throttle openings (35 degrees (absolute) or greater), if the driver pumps 


NASA, p. 170 


41 The engine intake manifold is the source of vacuum used by the brake booster to provide power assist. The 
engine manifold produces less vacuum as the throttle is opened from idle. Braking when the throttle is open will 
have full power assist for the first application only. If the brake pedal is “pumped” the booster reserve vacuum will 
be depleted after the first few applications. 



Vehicle Information 


Brake Hold 
at Wide Open 
Throttle 


Engine 

Transmission 


Brake Pedal Force 
Required (lbs.) 

Veh. 

ID. 

Model 

Trim 

Line 

1MY 

Config. 

Displacement 

Fwd Speeds 

Brake Pedal 
Single or 
Double 
Linkage 

Full 

Vacuum 

No 

Vacuum 

ID 

CAMRY 

SE 

2002 

V6 

3.0L 

4 

29.8lbs. 

u 

2D 

CAMRY 

XLE 

2002 

L4 

2.4L 

4 

3D 

CAMRY 

LE 

2001 

L4 

2.2L 

3 

V C5 


1H/.J 

4D 

CAMRY 

SE 

2007 

L4 

24L 

5 

\ s 

24.9 

193.0 

5D 

CAMRY 

LE 

2006 

14 

2.4L 

5 


154 

234.3 

6D 

CAMRY 

LE 

2007 

14 

24L 

5 

s\ 

25.3 

138.1 

7D 

CAMRY 

XLE 

2005 

14 

2.4L 

5 

D 

29.8 

167.1 

8D 

CAMRY 

XLE 

2001 

V6 

3.0L 

4 

s 

32.5 

158.1 

9D 

CAMRY 

LF. 

2005 

V6 

3.0L 

5 

D 

43.6 

268.2 

10D 

CAMRY 

LE 

2007 

V6 

3.5L 

6 

S 

30.9 

217.8 

1 ID 

CAMRY 

XLE 

2005 

V6 

3.0L 

5 

S 

25.7 

236.0 

i?r 

fAMRY 

YI F 

7007 

VA 

3 M 

6 


77 1 

1 JR f\ 
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Brake Test Data Table : http://www.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-Toyota_vehicle_characterization.pdf, p. 34 
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OUR ANALYSIS OF TOYOTA’S SOFTWARE 


13 chapters 

■ Vehicle code analysis 

+1 summary 

■ Case-specific analysis 

>750 pages 
+ appendices... 



That’s a GRANDE coffee! 
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“Highly Confidential” 

■ Even I don’t have a copy of my expert report! 

“Source Code Protective Order” 

■ The contract I signed to see the code is also secret! 


BUT a transcript of my testimony is around... 

■ Try “bookout toyota barr” 
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TEST SPACE EFFECTIVELY INFINITE 


Lots of ways for the software to malfunction 
And a malfunction can begin in lots of states 

■ Precise timing of events 

■ Internal software states 

■ Vehicle operating states 

Cruise on or off ? 

Accel at 5% or 50%? 

Failing 0 2 sensor? 

■ Driver reactions 

36 Copyright 2014 Barr Group. All rights reserved. 
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IN-VEHICLE TESTING 


2005 & 2008 Camry 

■ Fault-injection test 

■ Dynamometer 


Defects confirmed 

■ Gaps thru fail safes 
And a defect in one! 



■ Loss of throttle control 

Violation of a NHTSA safety standard 
Via a single point of failure (a bit flip!) 
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Caveat : Not of the actual dynamometer or test vehicle. 









TOYOTA’S TESTING 
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Each model “goes through a driving test for over 
400,000 miles. ... In this we have never confirmed 
an instance of unintended acceleration - Toyota 

■ BUT first 4,000 buyers do more testing in first week 
In more cars in more weather with more drivers etc. 


U.S. fleet of 2002-2007 Camrys: ~1 billion hrs/yr! 

- NASA, Appendix A, FN 24 
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Testimony : http://democrats.energycommerce.house.gov/sites/default/files/documents/lnterview-Ogawa-Kishi-Toyota-2010-3-18.pdf 





THE JURY VERDICT 


#KillerApps 


Damage award 

■ Toyota to pay Mrs. Bookout: $1.5M 

■ Toyota to pay Mrs. Schwarz’ estate: $1.5M 

Punitive finding 

■ Toyota acted with: “reckless disregard” 

Toyota settles acceleration lawsuit after $3- 
million verdict 

Toyota heads off punitive damages after a $3-million jury verdict pointed to software defects 
in a fatal crash. The case could fuel other sudden acceleration lawsuits. 
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Source : http://www.latimes.com/business/autos/la-fi-hy-toyota-damages-20131026,0,1605124.story 
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TOYOTA LITIGATION SUMMARY 


Plaintiff 

Court 

My Role 

Status 

Amount 

Saylor 

CA 

- 

settled Feb Ml 

$10M 

Van Alfen 

U.S. 

report Jul M2 

settled Dec M2 

private 

U.S. Class 

U.S 

report Jul M2 

settled Dec M2 

up to $1.5B 

Bookout 

OK 

testimony Oct ’13 

verdict Oct M3 

$3m + ?? 

Vance 

WV 

retained 

settled Dec M3 

private 

St. John 

U.S. 

report Apr ’13 

in talks Dec M3 

* 

Canada Class 

ON 

retained 

settled Mar M4 

~$150M 

Criminal 

U.S. 

- 

settled Mar M4 

$1.2B 

new cases 

various 

ongoing 

still being filed i 

$3bTM) 


* One of approximately 400 injury cases in settlement talks now v 

40 Copyright 2014 Barr Group. All rights reserved. Yet no remedy. 
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SAFETY FUTURE 


#KillerApps 
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THE SOFTWARE SAFETY LANDSCAPE 


Voluntary Standards 


IEC 61508 


AS026262 



Regulation/Oversight 



Litigation? 
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AUTOMOTIVE SAFETY 
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Modern vehicles are networks of computers 

■ Brake-by-wire, collision avoidance, etc. emerging... 


Tmtporteaian toon) 

SffClAI ukw >01 



The Safety 

Promise and Challenge 
of Automotive Electronics 


INSIGHTS FROM UNINTENDED ACCELERATION 


“FAA exercises far greater oversight of the verification and 
validation of designs and their implementation” than NHTSA. 

“NHTSA does not set its own design and implementation 
standards, nor does it demand that manufacturers follow 
third-party standards to guide design, development, and 
evaluation processes such as testing of software code” 


NATIONAL RESEARCH COUNCIL 
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TRB Special Report 308 : http://www.nap.edu/catalog.php?record_id=13342 
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HOW DO WE MAKE OUR SYSTEMS SAFER? 


No quick fix 

But certainly NOT... 

■ “It can’t be software” 

Sunshine is needed 

■ Informed oversight 

■ Less code confidentiality 



45 


Copyright 2014 Barr Group. All rights reserved. 


*13 BARR 

)J| group 







IMAGINE A WORLD... 


#KillerApps 


What if you could wave a magic wand? 

“Self-driving cars and smart highways for all” 


Road train 


CUD 


Independent 

vehide 



cr 


Following 

vehicles 


Laad 

vehicle 


tECD 


QHD 


Everyone is safer-on average-in and around cars! 

Accidents now caused by engineering mistakes 

Better/safer drivers lose advantages 

•1 
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WHEN WILL IT BE SAFE? 


#KillerApps 



http://www.barrgroup.com/killer-apps/ 
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